PoE: A Panel of Experts for Generalized Automatic Dialogue Assessment

Chatbots are expected to be knowledgeable across multiple domains, e.g. for daily chit-chat, exchange of information, and grounding in emotional situations. To effectively measure the quality such conversational agents, a model-based automatic dialogue evaluation metric (ADEM) is perform well domains. Despite significant progress, existing ADEMs tend only on data that similar its training (overfit domain). This calls domain-generalized can assess dialogues different characteristics. this end, we propose Panel Experts (PoE), multitask network consists shared transformer encoder collection lightweight adapters. The captures general knowledge while each adapter specializes one specific domain serves as expert. validate idea, construct high-quality multi-domain dataset leveraging augmentation pseudo-labeling. PoE comprehensively assessed 16 datasets spanning wide range It achieves state-of-the-art performance terms mean Spearman correlation over all datasets. exhibits better zero-shot generalization than ability easily adapt new domains with few-shot transfer learning.